next-gen inference hardware
Mixed-Precision Quantization for Next-Gen Inference Hardware
Compute requirements at the edge, such as the resource footprint of endpoints/edge devices & performance KPIs of Internet of Things applications make optimization of Deep Neural Networks (DNNs) critical for deployment. By deployment we assume that the Deep Learning (DL) model gets fixed in all aspects (e.g., task definition, dataset, training pipeline, etc.) so it can speed up model inferencing presuming the rest of the flow is equal. There are two main objectives for model optimization: reduction of the model size and decreasing computational complexity and both are highly correlated with each other. For example, reduction of the model size in many cases leads to less memory consumption and thus latency reduction, which, in turn, improves performance. Nowadays, many optimization methods have been proposed to tackle both objectives. Perhaps, uniform quantization is one of the simple methods to do that because it leads to a small accuracy drop, substantial reduction in the model size, and significant performance speedup can be applied even without retraining or fine-tuning the optimizing model.